A kernel method for canonical correlation analysis

1. Canonical correlation analysis (CCA) is a technique to extract common features from a pair of multivariate data. In complex situations, however, it does not extract useful features because of its linearity. On the other hand, kernel method used in support vector machine (Vapnik, 1998) is an efficient approach to improve such a linear method. In this study, we investigate the effectiveness of applying kernel method to the canonical correlation analysis. 2. Suppose there are pairs of multivariates x and y. We want to find transformations of x and y which extract features commonly embeded in x and y. CCA can find linear transformations which maximize the correlation coefficients between extracted features. However, if the embedding of common features is nonlinear or the relation between features is not Gaussian, CCA sometimes fails to find appropriate features. 3. In this presentation, we propose the kernel canonical correlation analysis (KCCA). Basic idea is as follows: First, the original pair of multivariate variables x and y are mapped into some high demensional Hilbert spaces Hx and Hy respectively, i.e., φx(x) and φy(y). After that, we find linear transformations of φx(x) and φy(y), which can find more complex transformation than CCA. 4. However, there are two problems to be solved, especially because of the dimensionality d of the Hilbert spaces. One is that it is an ill-posed problem if d is larger than sample size. The other is that time complexity to calculate φx(x) and φy(y) may be large. Let a and b be linear coefficients for φx(x) and φy(y) here. 5. In order to solve the former problem, we introduce a quadratic regularization term η(a + b)/2 into the cost function of CCA. By the quadratic regularization, it follows that a can be written by a weighted sum of φx(xi) where xi is the i-th sample, and b can be written by a weighted sum of φy(yi). Therefore, a · φx(x) = ∑ i αiφx(xi) · φx(x). This fact enables us to use a “kernel trick”: Let k(z, w) be a kernel function which is symmetric and positive definite, then there exists φz(z) and k(z, w) = φz(z) · φz(w). Using a kernel, we can calculate φx(xi) · φx(x) directly without knowing φ. This means the complexity problem of calculation is solved as well, because we do not need to calculate φ anymore. 6. Consequently, we obtain KCCA: 1. Calculate matrices of kernels Kx = (kx(xi, xj)) and Ky = (ky(yi, yj)), where kx and ky are kernels. 2. Solve the generalized eigen problem, Mβ = λLα and Mα = λNβ, where M = (1/n)K x JKy, L = (1/n)K T x JKx + (η/λ)Kx, N = (1/n)K T y JKy + (η/λ)Ky and J = I − 1 1, 1 = (1, . . . , 1) . 7. Some simulation results show KCCA recovers nonlinear relation between multivariate variables effectively. However, the performance is sensitive to the regularization constant η. If η is too small, the solution is unstable and there are many features achieves large correlation coefficient, which makes difficult to choose relevant features. On the other hand, if η is too large, all the correlation coefficients tend to be small. Currently, we choose η by cross validation test. 8. We have proposed the extension of CCA by introducing the kernel method. Canonical correlation analysis includes Fisher discriminant analysis (FDA) as a special case. Mika et al. (1999) has proposed the kernel method for FDA, which is also included in our study.